Toward Experimental Evaluation of Subsystem Classification Recovery Techniques

Several reverse engineering techniques classify software system components into subsystems. These techniques are designed to discover such classifications when the classifications are unknown. The techniques are tested and evaluated, however, by matching the classifications they recover against expected classifications. Several such techniques may be compared by experimentally evaluating their performance on the same set of software systems. Two things are needed to ensure experiment repeatability: (1) a set of “real-world” software systems whose expected subsystem classifications are known, and (2) an objective criterion to quantitatively determine the similarity of subsystem classifications. This paper contributes to both needs by identifying a set of widely used and easily accessible software systems whose modular decomposition either is documented or can be easily inferred from their design philosophy, and by presenting a measure to quantitatively determine the congruence between hierarchical subsystem classifications.